Hey @oquenchil, do you think this might make it on time for Bazel 5 (#14013)? This is mostly the same patch as before, with the change you asked me to make. Thanks!

Sorry, I need to carefully review this change and see that it doesn't affect anything negatively internally. I haven't had time to do so yet.

Thanks for being so patient.

This looks good and internally the tests pass with minor modifications that I can take care of. But I'm still trying to wrap my head around all this, so please bear with me.

Unless running under the execroot, the transitioned solib directory may not be available at all; the right files will be present under the default solib directory, so it should be on the rpath.

When you say the execroot, you mean the repository root where you run bazel build, do you mean where actions themselves run or something else?

As I understand it, if you move the location of the executable, ORIGIN changes, but the code still works regardless of where you move the binary, doesn't it? So where is the solibdir when you have $ORIGIN/../_solib_k8/? You go one directory up from where you are running the binary and then there should be a directory called solib_k8? Do you have to put it there manually? Is it part of runfiles and you just have to copy the runfiles?

That's for the default solibdir. Why doesn't the same thing apply for the transitioned solibdir, i.e.: ../_solib_k8/../../../k8-fastbuild-ST-? If you placed the default solibdir there manually, why not this one as well? If the default solibdir was in runfiles and you copied runfiles, how come that the transitioned solibdir isn't in runfiles too?

I realize that it should be the other way around where I'm answering the questions but since you have a better understanding of what the actual problem is I thought you could help.

Thanks!

I should be the one to thank you – I'm sorry I've been nagging you about it, and I really appreciate your taking the time to work on this. I'm happy to help any way I can; if you'd like to have a chat over this, I can try and demo what I'm seeing – please just let me know.

There seems to be a difference between the layout in local execution (i.e. where bazel build is run: this is what I meant by 'under the execroot') and the directory structure where actions are run remotely. $ORIGIN is the directory where the executable resides, and the structure of the runfiles dir is such that there's one solib directory there, where all the correctly transitioned SOs are available. When depending on transitioned libraries, rpath entries are added that go all the way up to k8-fastbuild-ST-... directories for other configurations. Locally, they do exist and everything works well. In remote execution (and deployment by copying the executable and its runfiles), we only have the runfiles directory, so the only rpath entry that can be actually used is the one that points to the default solib directory. But we don't get an rpath entry pointing there unless we're linking a non-transitioned library. If we don't, all the libraries that Bazel put there can't be found.

Indeed, the trigger for filing the original issue was a cc_test that always failed in remote execution, but always passed locally. I found that it depended only on transitioned libraries – so the default solib dir wasn't on the rpath, whereas that's where all the SOs were in RE. I put together a gist that I hope demonstrates this: https://gist.github.com/quval/5077b2edc902e9dc0bcad2311a245ffa.

Alright, understood, thank you very much for the great contribution. I will merge now.

@ulfjack Have you never bumped into this issue?